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Summary 


Problem  and  Background 

The  challenges  facing  Antisubmarine  Warfare  (ASW)  training  in  the  closing  years  of  this 
decade  are  greater  than  at  any  time  since  the  early  days  of  World  War  11.  Conditions  since  the  end 
of  the  Cold  War,  and  those  expected  throughout  the  next  decade,  impose  additional  complexity  on 
maintaining  our  ASW  superiority.  Russian  nuclear  submarine  technology  continues  to  improve 
and  advanced  submarines  continue  to  be  built  and  delivered  to  their  fleet.  Concurrently,  the 
proliferation  of  improved  diesel  submarine  technology  to  many  Third  World  nations  requires  that 
our  ASW  forces  also  be  capable  of  conducting  operations  in  the  vastly  different  littoral  regions. 

The  training  challenge  is  two-fold:  (1)  retaining  the  capability  to  detect  and  prosecute  nuclear 
submarines,  and  (2)  expanding  our  current  capability  against  diesel  submarines  of  the  Third  World. 
When  coupled  with  dramatic  reductions  in  ASW  training  resomces,  including  at-sea  training,  this 
historic  change  compels  the  development  of  training  for  skills  learned  previously  on-the-job  and 
for  skills  required  in  new  environments  for  both  sensor  operators  and  tacticians. 

The  Interactive  Multisensor  Analysis  Training  (IMAT)  system  was  developed  to  address  post 
Cold  War  ASW  training  requirements.  Specifically,  IMAT  is  designed  to  teach  the  complex 
conceptual  knowledge  and  cognitive  and  procedural  skills  required  to  reason  about  the 
interrelationships  among  the  operating  modes  of  target  submarines,  the  environmental  variables 
that  affect  sound  transmission,  and  the  sensor  systems  used  for  detection  and  localization.  In 
addition,  IMAT  provides  extensive  training  on  submarine  acoustics  and  interpretation  of  sensor 
system  data.  The  IMAT  approach  to  training  is  based  on  recent  advances  in  cognitive  psychology 
and  instructional  technology.  It  combines  cognitive,  analytic,  and  curriculum  design  technology 
with  advanced  computer-based  graphics  and  programming  technology  to  present  state-of-the-art 
training. 

Objectives 

The  objectives  of  this  effort  are  (1)  to  evaluate  the  application  of  the  IMAT  system  in  the  Sonar 
Technician  Surface  (STG)  “A”  school  with  respect  to  performance  on  factual,  comprehension,  and 
cognitive  skill  test  items,  student  motivation,  and  instructional  design;  (2)  to  compare  student 
performance  on  STG  “A”  IMAT  lessons  with  student  performance  on  comparable  Aviation 
Systems  Warfare  Operator  (AW)  A  school  IMAT  lessons;  (3)  to  evaluate  application  of  the  IMAT 
system  in  the  Fleet  Aviation  Specialized  Operations  Training  Group  Pacific  Tactical  Training 
Course  (TTC)  at  Barbers  Point;  and  (4)  to  compare  graduates  of  the  IMAT  TTC  with  qualified  fleet 
tactical  officers  and  pilots  on  tests  of  knowledge  of  acoustical  oceanography  and  tactical  problem 
solving. 

Method 

The  STG  “A”  IMAT  application  was  evaluated  on  (1)  student  performance  on  three  types  of 
end  of  unit  test  items  (fact  items,  comprehension  items,  and  cognitive  skill  items),  (2)  student 
motivation,  and  (3)  quality  of  instructional  design.  Seventy  one  STG  students  trained  with  the 
IMAT  system  were  compared  with  90  STG  students  trained  with  the  standard  instruction  used  in 
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the  “A”  school  prior  to  the  introduction  of  IMAT,  and  with  46  students  from  the  AW  “A”  School 
who  had  also  been  trained  with  the  IMAT  system. 


For  the  TTC  evaluation,  59  students  were  administered  a  paper  and  pencil  test  that  included 
both  knowledge  items  and  scenario  based  cognitive  skills  items.  They  were  tested  upon  entering 
the  school  and  after  completing  the  course  in  a  standard  pre-post  design.  The  scores  of  the  TTC 
graduates  were  also  compared  with  the  performance  of  21  qualified  fleet  tacticians  and  pilots  on 
the  same  testing  instrument. 

Results 

The  STG  IMAT  students  scored  higher  on  all  item  types  than  the  STG  Standard  Instruction 
students.  Further,  the  instructional  design  analysis  showed  that  instructional  quality  was  superior 
in  the  STG  IMAT  lessons.  Unfortunately,  the  motivation  data  for  the  STG  Standard  Instruction 
students  was  either  lost  or  not  collected.  However,  motivation  for  the  STG  IMAT  students  was  high 
and  compared  favorably  to  previous  motivation  data  for  AW  IMAT  students. 

AW  IMAT  students  scored  higher  on  all  item  types  than  the  STG  IMAT  and  STG  Standard 
Instruction  students.  Again,  the  instructional  design  analysis  showed  that  the  AW  IMAT  instruction 
was  higher  in  overall  quality  than  the  materials  for  the  other  two  groups.  Also,  both  the  STG  and 
AW  IMAT  groups  scored  relatively  better  on  cognitive  skill  and  comprehension  items  compared 
to  fact  items  than  STG  Standard  Instruction  students. 

The  findings  for  the  TTC  course  evaluation  showed  that  TTC  students  had  significant  gains 
from  pretest  to  posttest  for  knowledge  and  scenario  test  items.  Further,  in  the  comparison  with 
qualified  fleet  personnel,  TTC  graduates  scored  higher  than  the  qualified  fleet  personnel  on  both 
scenario  and  knowledge  items. 

Conclusions 

Several  conclusions  can  be  drawn  from  this  study.  First,  the  IMAT  approach  to  training,  based 
on  computer-generated  dynamic  displays  and  instructional  design  and  delivery  strategies  designed 
to  enhance  cognition,  produces  substantial  gains  in  performance.  This  is  especially  true  for  the 
explicit  IMAT  goal  of  teaching  complex  knowledge  and  cognitive  skills.  Second,  the  IMAT  system 
emphasis  on  adhering  to  the  principles  of  high  quality  instructional  design  was  supported  in  the 
instructional  design  analyses.  The  differences  in  design  quality  among  the  STG  Standard 
Instruction,  STG  IMAT,  and  AW  IMAT  groups  undoubtedly  contributed  to  the  observed 
performance  differences.  Finally,  the  TTC  findings  show  that  IMAT  can  bring  course  graduates  up 
to  the  level  of  fleet  qualified  personnel  on  both  fact  and  scenario  tests.  Overall,  the  results  of  the 
STG  and  TTC  implementations  show  that  IMAT  is  a  highly  effective  training  system  that  offers  a 
viable  solution  for  many  of  the  training  requirements  and  challenges  faced  in  the  post  cold  war 
world. 
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Introduction 


Problem  and  Background 

The  challenges  facing  Antisubmarine  Warfare  (ASW)  training  in  the  closing  years  of  this 
decade  are  greater  than  at  any  time  since  the  early  days  of  World  War  H.  During  the  Cold  War, 
ASW  mission  requirements  (and  the  training  designed  to  support  those  requirements)  were  driven 
by  the  need  to  combat  Soviet  nuclear  submarines  in  open  ocean  environments.  The  prevailing 
ASW  strategy  was  designed  to  detect  and  prosecute  enemy  submarines  at  long  ranges,  and 
operations  were  most  often  conducted  by  single  ASW  units  with  minimal  coordination  with 
outside  assets.  The  relatively  benign  deep  ocean  environment  and  nearly  exclusive  focus  on  the 
Soviet  threat  resulted  in  the  development  of  effective  and  well  practiced  sensor  and  weapons 
tactics.  ASW  training  mirrored  that  relatively  narrow  focus,  and,  combined  with  frequent  real- 
world  encounters,  was  quite  effective  in  producing  competent  ASW  operators  and  tacticians. 

Conditions  since  the  end  of  the  Cold  War,  and  those  expected  throughout  the  next  decade, 
impose  additional  complexity  on  maintaining  our  ASW  superiority.  Russian  nuclear  submarine 
technology  continues  to  improve  and  advanced  submarines  continue  to  be  built  and  delivered  to 
their  fleet.  Concurrently,  the  proliferation  of  improved  diesel  submarine  technology  to  many  Third 
World  nations  requires  that  our  ASW  forces  also  be  capable  of  conducting  operations  in  the  vastly 
different  littoral  regions. 

Littoral  environments  introduce  added  difficulty  in  optimally  employing  onboard  sensors  and 
weapons  tactics.  Coastal  areas  with  shallow  water,  complex  bathymetry  and  bottom  topography, 
heavy  shipping,  and  highly  variable  environmental  conditions  impose  significant  restrictions  on 
traditional  deep  water  tactics.  The  diesel  submarine  operating  in  home  waters  has  the  additional 
advantage  of  familiarity  with  environmental  anomalies,  slow  speed  operation,  and  support  from 
own  nation  defense  systems. 

The  training  challenge  has  thus  become  two-fold;  retaining  the  Cold  War  capability  to  detect 
and  prosecute  nuclear  submarines  while  expanding  our  current  capability  against  diesel 
submarines  of  the  Third  World.  When  coupled  with  dramatic  reductions  in  ASW  training 
resources,  including  at-sea  training,  this  historic  change  compels  the  development  of  different 
approaches  to  training  for  both  sensor  operators  and  tacticians. 

Traditional  ASW  Training 

Sensor  Operators.  In  the  past,  the  ASW  training  has  been  based  on  a  balance  between 
schoolhouse  and  operational  training.  To  become  journey  level  operators,  students  learned  the 
basics  in  schoolhouse  training  and  then  received  extensive  on-the-job  training  through  supervised 
practice  by  experienced  fleet  crews  at-sea.  The  schoolhouse  training  focused  on  memorizing  facts, 
procedures,  and  large  databases  of  threat  intelligence  parameters.  Students  were  not  taught  to  think 
about  and  relate  the  underlying  physics  of  threats,  the  environment  and  the  sensor  systems.  This 
approach  resulted  in  graduates  who  could  answer  specific  factual  questions  based  on  memorized 
information  and  who  could  perform  procedures  but  who  had  difficulty  in  applying  knowledge  and 
principle  to  solve  problems  in  operational  situations.  Great  reliance  was  placed  on  a  substantial 
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amount  of  at-sea  experience  to  transfer  the  knowledge  and  skill  gained  in  formal  training  to 
operational  competency. 

Tacticians.  For  years,  frequent  successful  detection  and  encounters  with  Soviet  submarines 
provided  fleet  tacticians  the  best  kind  of  on-the-job  training  in  ASW  tactics.  Schoolhouse  tactical 
training  was  not  extensive  and  focused  on  basic  knowledge,  procedures,  and  crew  coordination. 
For  example,  classroom  training  typically  emphasized  the  principles  of  oceanography  and  the 
effects  of  the  ocean  environment  on  sound  transmission  but  provided  little  in  depth  practice  in 
applying  environmental  information  to  tactical  planning  and  mission  execution.  The  development 
of  tactical  planning  skills  was  acquired  on-the-job.  A  recent  study  (Reynolds,  Wetzel-Smith,  Ellis, 
&  Wulfeck,  in  press)  of  tacticians  and  pilots  in  the  Maritime  Patrol  Aviation  community  examined 
the  relationship  between  job  conditions  and  tactical  planning  skills  as  measured  by  a  scenario 
based  planning  skills  test.  The  results  showed  that  higher  levels  of  work  site  training  and  technical 
support  produced  higher  test  scores.  Officer  ratings  of  the  training  received  on  the  job  and  self 
proficiency  ratings  were  also  positively  related  to  test  scores.  However,  test  scores  were  generally 
low.  Mean  scores  for  the  highest  performers  were  below  60  percent.  In  addition,  the  amount  of 
operational  experience  reported  was  low  and  was  not  associated  with  test  performance.  This 
finding,  in  conjunction  with  the  overall  low  scores  found  for  the  planning  skills  tests,  indicates  that 
current  job  conditions  alone  may  not  be  adequate  to  support  proficiency  in  tactical  planning  and 
mission  execution. 

Post  Cold  War  ASW  Training  Requirements 

There  is  now  a  broad  consensus  across  all  four  ASW  training  communities  that  the  combined 
change  in  threat,  environment,  and  operating  circumstances  requires  a  new  training  approach. 

Current  (leet  ASW  practitiopers  receive  little  at-sea  operational  practice  against  non- 
cooperative  submarines.  As  a  result,  an  operators  ability  to  detect,  classify,  and  track  submarine 
targets  will  he  directly  related  to  the  quality  of  initial  training,  and  to  bow  well  those  skills  are 
maintained  and  increased  throughout  a  career.  The  same  will  be  true  for  the  tactician  whose 
responsibility  for  planning  and  directing  ASW  missions  requires  a  far  more  qualitative 
understanding  of  the  environment,  own  and  other  systems  and  tactics,  as  well  as  target  sensor 
capability,  tactics,  and  weapon  systems.  To  achieve  this  level  of  competence,  the  limited  hours 
available  for  both  schoolhouse  and  at-sea  training  must  be  used  to  provide  both  a  solid  conceptual 
understanding  of  the  complex  tasks  that  individuals  and  crews  must  perform  as  well  as  the 
opportunity  to  practice  those  tasks  under  varying  real-world  conditions. 

Achieving  these  goals  will  require  substantive  changes  in  the  execution  of  schoolhouse  and 
operational  training.  ASW  training  in  the  1990s  must  see  dramatic  modifications  to  passive 
acoustic  analysis  training  and  substantial  expansion  of  training  in  active  sonar,  radar,  and 
electromagnetic  sensor  systems  to  enable  operators  to  contribute  effectively  in  a  multi-sensor 
approach  to  submarine  prosecution.  Further,  the  complexity  of  the  acoustic  environment  in  the 
littoral  regions  requires  a  substantial  increase  in  knowledge  of  the  effects  of  ocean  bathymetry  on 
acoustic  energy  transmission  and  how  that,  in  turn,  affects  sensor  selection  and  placement.  These 
training  requirements  can  only  be  met  through  principle-based  application  of  training  technologies, 
that  can  provide  conceptual  knowledge  and  high  fidelity  experience  to  offset  the  lack  of  at-sea 
practice. 
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The  Interactive  Multisensor  Analysis  TVaining  (IMAT)  System 

Overview 

The  Interactive  Multisensor  Analysis  Training  (IMAT)  system  was  developed  to  address  post 
Cold  War  ASW  training  requirements.  Specifically,  IMAT  is  a  classroom  based  approach  to 
training  that  is  designed  to  teach  the  complex  conceptual  knowledge  and  cognitive  and  procedural 
skills  required  to  reason  about  the  interrelationships  among  the  operating  modes  of  target 
submarines,  the  environmental  variables  that  affect  sound  transmission,  and  the  sensor  systems 
used  for  detection  and  localization.  In  addition,  IMAT  provides  extensive  training  on  submarine 
acoustics  and  interpretation  of  sensor  system  data. 

The  IMAT  approach  to  training  is  based  on  recent  advances  in  cognitive  psychology  and 
instructional  technology.  It  combines  cognitive  analytic  and  instructional  design  technology  with 
advanced  computer-based  graphics  and  programming  technology.  The  result  is  a  dynamic 
graphical  interface  integrated  with  state-of-the-art  instructor  and  student  guides,  which  provides 
the  traditional  classroom  instructor  with  a  capability  to  effectively  teach  complex  cognitive 
concepts  and  skills.  In  the  past,  achieving  this  capability  has  been  hampered  by  limitations  in 
cognitive  task  analysis,  and  particularly  in  cognitive  models.  In  addition,  limitations  in  computer 
capabilities  have  precluded  the  development  of  cause  and  effect  representations  of  highly  complex, 
multi-modal  tasks  that  are  required  of  expert  practitioners.  Recent  developments  in  cognitively- 
based  training  design  have  demonstrated  that  models  of  physical  phenomena  can  be  integrated  with 
high  resolution  graphics  to  demonstrate  the  interactive  relationships  of  threat,  environment,  and 
system  for  operator  training,  and  interactions  of  multiple  sensor  systems  for  tactician  training.  The 
IMAT  system  extends  that  technology  in  the  traditional  classroom  environment,  with  specific 
emphasis  on  cognitive  design  models  that  account  for  knowledge  structure  interrelationships,  to 
provide  cause  and  effect  training  for  decision  making  with  multiple  system  input,  and  to  allow 
visualization  of  interactive  spatial  relationships  among  operators,  sensors,  and  other  platforms. 

Specifically,  for  sensor  operators,  IMAT  apprentice  training  is  presented  in  a  mission  context 
with  a  substantial  emphasis  on  the  interactive  relationship  among  environmental  factors,  threat 
behavior,  and  sensor  system  capabilities  and  constraints.  Because  sensor  operators  at  the 
apprentice  level  have  limited  backgrounds  in  submarine  operations,  the  physics  underlying  energy 
transmission,  and  in  operating  complex  sensor  systems,  establishing  a  good  conceptual 
understanding  of  the  complicated  interactions  that  occur  in  the  real  world  is  the  essential 
foundation  for  learning  effective  sensor  operation.  Students  are  taught  to  understand  that  the  enemy 
submarines  mission  will  largely  dictate  its  operating  mode  including  course,  speed,  and  depth  of 
operation,  and  that  the  operating  mode  of  the  submarine  defines  its  vulnerabilities  to  onboard 
acoustic  and  electromagnetic  sensors.  They  are  further  taught  how  the  relative  complexity  of  the 
ocean  environment  will  impact  detection  ranges,  search  rates,  and  contact  duration.  Students  learn 
to  reason  through  these  interactions  in  a  cause  and  effect  learning  process.  Multi-dimensional 
interactions  are  displayed  visually  and  the  IMAT  instruction  provides  qualitative  explanations  for 
the  interactions  that  occur.  IMAT  presents  demonstrations  of  varying  outcomes  based  upon 
changes  in  the  threat  or  environment  to  promote  the  development  of  the  principle-based  knowledge 
critical  for  adaptations  to  the  variations  presented  in  real  world  situations.  Finally,  tests  of  student 
proficiency  for  IMAT  training  include  questions  that  require  problem  solving  and  understanding 
causal  relationships  in  addition  to  the  traditional  fact  recognition  and  procedural  process  items. 
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IMAT  training  for  tacticians  is  similar  to  the  training  for  sensor  operators.  Establishing  a  good 
conceptual  understanding  of  the  complicated  interactions  that  occur  in  the  real  world  is  an  essential 
foundation  for  learning  effective  platform  and  sensor  employment.  Therefore,  IMAT  presents 
tactical  training  in  a  mission  context  with  a  substantial  emphasis  on  the  interactive  relationships 
among  environmental  factors,  threat  behavior,  and  sensor  system  capabilities  and  constraints. 
Although  applied  to  different  task  responsibilities  than  the  sensor  operators,  these  critical 
relationships  form  the  basis  for  developing  tactical  skills.  Like  operators,  IMAT  trained  tacticians 
are  taught  to  understand  that  the  submarine’s  mission  will  largely  dictate  its  operating  mode, 
including  courses,  speeds,  and  depths  of  operation  and  the  operating  mode  of  the  submarine  makes 
it  vulnerable  to  detection  by  onboard  sensors  and  which  are  best  suited  for  employment.  IMAT 
elaborates  how  the  relative  complexity  of  the  ocean  and/or  atmospheric  environment  impacts 
detection  ranges,  search  rates,  and  anticipated  contact  duration,  and  then  teach  tacticians  the 
knowledge  required  to  employ  their  platforms  and  direct  their  combat  sensor  teams  to  optimize 
search  rates  and  probabilities  of  detection  within  both  temporal  and  search  area  constraints.  As  in 
operator  training,  IMAT  teaches  tacticians  how  to  reason  through  these  interactions  in  a  cause  and 
effect  learning  process  using  displays  that  demonstrate  multi-dimensional  interactions  with 
varying  outcomes  visually.  IMAT  instruction  provides  qualitative  explanations  for  interactions  that 
occur  with  the  goal  of  developing  principle-based  knowledge  critical  for  adapting  to  the  various 
real  world  situations.  Proficiency  assessment  for  tacticians  includes  tactical  scenario  problems  as 
well  as  facts  and  causal  relationship  questions. 

Research  Background 

The  IMAT  system  integrates  several  areas  of  research  on  cognition  and  instruction,  including, 
graphical  techniques  to  promote  visualization  of  invisible  phenomena  in  science  teaching, 
elaborated  explanations,  contextualized  or  anchored  instruction,  and  instructional  sequencing.  The 
following  sections  briefly  summarize  portions  of  this  work. 

Scientific  Visualization.  Scientific  visualization  has  traditionally  been  used  by  scientists  to 
explore  phenomena  and  to  communicate  with  other  scientists  (Bryson,  1994).  When  used  for 
presentations,  researchers  select  data  sets,  transform  them,  and  then  turn  them  over  to  specialized 
graphic  artists  to  develop  images  and  animation.  However,  the  end  products  of  this  process  have 
not  been  designed  for  laymen  or  students.  IMAT  aims  to  bring  this  technology  into  specialized 
technical  training. 

Research  support  for  scientific  visualization  as  a  training  strategy  comes  from  the  literature  on 
instructional  media.  Both  static  and  dynamic  graphic  displays  have  been  shown  to  facilitate 
teaching  of  scientific  concepts  (Baek  &  Layne,  1988;  Dwyer,  1972;  Gropper,  1966;  Lumsdaine, 
Sulzer,  &  Kopstein,  1961;  Rieber,  1990;  Rigney  &  Lutz,  1975;  Park  &  Gittelman,  1992,  Wetzel, 
Radtke,  &  Stem,  1994):  Levie  and  Lentz  (1982)  in  a  meta-analysis  of  illustrated  text  studies 
concluded  that  learning  and  retention  is  facilitated  by  illustrations,  if  the  illustrations  are  directly 
related  to  the  text  Park  and  Gittelman  (1992)  found  that  subjects  trained  with  dynamic  graphics 
performed  better  on  electronic  troubleshooting  problems  that  those  trained  with  static  displays. 
White  (1984)  used  animated  computer  graphics  to  successfully  teach  the  basic  principles  of 
Newtonian  laws  of  motion  and  force.  IMAT  employs  a  computer  based  graphical  interface  to 
conceptual  models  of  real  world  phenomena  to  deliver  both  static  and  dynamic  graphics  in  a 
traditional  classroom  environment. 
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Elaborated  Explanations.  Providing  students  with  elaborated  explanations,  analogies,  etc. 
about  how  and  why  systems,  events,  and  phenomena  are  structured  and  function  has  been  shown 
to  facilitate  learning  and  retention.  Research  on  learning  skills  and  learning  from  text  has  shown 
that  elaborated  explanations  enhance  the  students  mental  models  and  increase  retention  (Konoske 
&  Ellis,  1991;  Mayer,  1989;  Smith  &  Goodman,  1982).  In  a  series  of  studies  of  learning  from 
scientific  text,  Mayer  (1989)  found  that  providing  students  with  a  conceptual  model  increased 
learning,  retention  and  transfer.  The  conceptual  models  in  his  instruction  used  both  text  and 
diagrams  to  highlight  major  objects  and  actions  and  the  causal  relations  among  them.  That  is,  the 
models  focused  on  how  and  why  systems  work.  Smith  and  Goodman  (1982)  studied  the  effects  of 
providing  elaborated  instructions  on  learning  and  performing  a  procedural  assembly  task  and 
found  that  instructions  containing  functional  information  resulted  in  fewer  errors.  Swezey,  Perez, 
and  AUen  (1991),  in  a  study  on  transfer  of  electromechanical  troubleshooting  skill,  found  that  some 
level  of  generic  structure  and  functional  knowledge  is  required  for  cross  domain  transfer.  The 
IMAT  system  uses  elaborated  explanations  throughout  the  instruction  to  (1)  clarify  complex 
relationships  such  as  those  among  water  temperature,  pressure  and  depth,  and  salinity;  (2)  provide 
comprehensive  feedback  for  practice  exercises;  and  (3)  describe  graphically  displayed  examples. 

Contextualized  Instruction  and  Instructional  Sequencing.  Contextualized  or  job  oriented 
instruction  has  been  found  to  be  more  effective  in  learning,  retention,  and  performance  than  topic 
oriented  instruction  (Cognition  &  Technology  Group  at  Vanderbilt,  1990;  Collins,  Brown,  & 
Newman,  1989;  Gofifard,  Heimstra,  Beecroft,  &  Oppenshaw,  1960;  Johnson,  1951;  Semb  &  Ellis, 
1994;  Shoemaker,  1960;  Steinemann,  Harrigan,  &  VanMatre,  1967; ).  Further,  within  a  job  context, 
mental  model  development  is  facilitated  by  teaching  students  to  reason  about  events  and 
phenomena  that  involve  several  interrelated  variables.  Proper  sequencing  may  play  an  important 
role  in  cognitive  skill  development.  While  early  research  on  sequencing  showed  that  with 
simplified  or  isolated  tasks,  different  sequences  of  instructional  events  made  little  difference,  more 
recent  research  and  theory  suggests  that  for  complex  tasks,  sequencing  strategies  may  have 
significant  effects.  For  example,  Reigeluth  and  Stein  (1983)  argues  that  beginning  instruction  with 
a  condensed  holistic  overview  of  a  task  domain  leads  to  better  learning  than  more  traditional 
sequences,  which  teach  isolated  topics  first  and  integrate  them  later.  More  recently,  extreme 
constructivist  approaches  to  instruction  (e.g.  Duffy  &  Jonassen,  1991)  argue  that  learners  should 
sink-or-swim  in  a  fully  elaborated  domain.  Merrill,  Li,  &  Jones  (1990)  also  argue  for  a  holistic 
approach  to  teaching  complex  domains,  but  include  moderate  structure  and  sequencing 
recommendations  in  their  approach.  Drawing  from  Reigeluth  and  Stein  (1983),  IMAT  begins  with 
a  simplified  overview  of  target,  environment,  and  sensor  system  relationships  in  the  context  of  the 
jobs  and  tasks  performed  by  operators  and  tacticians.  This  context  is  revisited  throughout  IMAT  to 
reinforce  the  reality  that  students  are  learning  to  do  a  job,  not  memorize  a  list  of  topically  related 
facts. 

Research  Questions 

In  each  of  these  areas,  little  experimental  work  has  been  done  on  the  extent  to  which  the 
findings  are  generalizable  to  instruction  delivered  using  simulation-  and  graphical-interface-based 
training  technologies.  Furthermore,  there  are  almost  no  larger  efforts  that  evaluate  the  integration 
of  these  approaches  into  an  overall  strategy.  The  current  effort  tests  the  hypothesis  that  the  IMAT 
system,  which  represents  an  integrated  combination  of  these  approaches,  offers  a  potent  learning 
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environment  for  promoting  acquisition  of  the  complex  knowledge  and  skills  involved  in  sensor- 
system  operation  and  tactical  planning.  Specifically,  this  report  documents  the  implementation  and 
evaluation  of  the  IMAT  system  in  the  Sonar  Technician  Surface  (STG)  “A”  School  at  the  Fleet 
Antisubmarine  Warfare  Training  Center  Pacific  in  San  Diego  (sensor  operators)  and  the  Tactical 
Training  Course  (TTC)  at  the  Fleet  Aviation  Specialized  Operations  Training  Group  Pacific  at 
Barbers  Point  (tacticians).  These  two  IMAT  applications  are  described  in  the  following  sections 
followed  by  the  specific  research  objectives  for  this  effort.  The  description  of  the  STG  “A”  school 
application  includes  a  review  of  the  first  IMAT  implementation  and  evaluation  in  the  Aviation 
Antisubmarine  Warfare  Operator  (AW)  “A”  school  in  Millington  (Ellis  &  Parchman,  1994) 
because  data  from  this  school  are  compared  with  STG  “A”  data  in  the  present  evaluation. 

Sensor  Operators-AW  “A”  and  STG  “A”  Schools.  The  first  IMAT  classroom 
implementation  was  in  the  AW  “A”  school  at  Naval  Air  Technical  Training  Center,  Millington,  TN. 
IMAT  replaced  the  existing  oceanography  unit  in  the  “A”  school  curriculum.  Although  the  basic 
classroom  configuration  and  environment  were  not  altered,  the  IMAT  unit  represented  a  significant 
change  from  the  existing  instruction.  The  unit  length  was  increased  from  26  to  58  class  periods  and 
there  were  extensive  changes  in  both  the  instructor  and  student  guides,  which  were  redesigned 
using  IMAT  criteria  to  provide  both  elaborated  explanations  and  more  contextualization. 
Furthermore,  the  IMAT  graphical  interface  was  used  to  provide  the  classroom  instructor  with  both 
static  and  dynamic  displays  of  a  important  concepts  and  relationships  including,  sound 
transmission  paths,  water  column  characteristics,  target  motion,  and  sound  propagation.  All  of 
these  changes  resulted  in  a  course  that  emphasized  learning  cognitive  skills  and  complex 
relationships  instead  of  memorizing  factual  information,  which  was  the  focus  of  the  existing  unit. 
A  formative  evaluation  of  this  application  (Ellis  &  Parchman,  1994)  found  increased  motivation 
for  IMAT  students  but  revealed  significant  problems  with  the  unit  test,  which  limited  the 
assessment  of  performance  differences.  The  unit  test  for  the  IMAT  instruction  was  very  similar  to 
the  test  for  the  oceanography  unit  that  IMAT  replaced.  The  formative  evaluation  showed  that  many 
IMAT  objectives  were  not  tested.  Further,  the  previous  course  test  items,  included  in  the  IMAT  unit 
test,  tested  factual  knowledge  but,  did  not  test  the  cognitive  concepts  and  skills  that  IMAT  was 
designed  to  teach.  Thus,  while  the  results  of  the  performance  assessment  showed  that  IMAT 
students  performed  as  well  as  previous  course  students  on  the  factual  items  common  to  both  tests, 
it  was  not  possible  to  compare  student  performance  on  the  more  complex  knowledge  and  skills.  As 
a  result  of  the  formative  evaluation  the  AW  “A”  school  IMAT  unit  test  on  oceanography  was 
extensively  revised.  The  revised  test  replaced  the  fact-based  test  in  the  AW  “A”  school  and  was 
also  used  as  the  primary  evaluation  instrument  in  the  STG  “A”  school  IMAT.  This  allowed 
comparisons  between  AW  and  STG  IMAT  students. 

The  AW  “A”  school  IMAT  oceanography  unit  was  adapted  for  implementation  in  the  STG  “A” 
school.  The  STG  “A”  revision  was  not  as  extensive  as  the  AW  “A”  schools.  Because  of  course 
length  constraints,  the  number  of  class  periods  (26)  was  not  increased  for  the  revised  course. 
Several  topics  were  eliminated  because  they  were  either  not  relevant  to  the  STG  rating  or  were 
taught  in  other  sections  of  the  course.  The  STG  instructor  guide  was  completely  revised  using  the 
AW  instructor  guide  as  a  basis  and  the  STG  student  guide  was  augmented  by  adding  portions  of 
the  AW  student  guide.  Duiing  instruction,  the  IMAT  trainer  was  used  by  the  classroom  instructor 
in  the  same  way  it  was  used  in  the  AW  unit.  The  STG  end  of  unit  test  was  comprised  of  relevant 
items  from  the  AW  end  of  unit  test. 
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Tacticians— Tactical  Training  Course  (TTC).  The  Tactical  Training  Course  (TTC)  was 
developed  to  provide  qualified  Patrol  Plane  Naval  Communicators  and  Patrol  Plane  Second  Pilots 
with  the  tactical  knowledge  and  skills  needed  to  perform  Tactical  Coordinating  Officer  (TACCO) 
and  Patrol  Plane  Commander  (PPC)  responsibilities.  The  TTC  was  a  new  course  and  did  not 
replace  any  existing  classroom  instruction.  The  course  emphasizes  tactical  oceanography  and 
practical  on-station  application  of  tactics.  The  IMAT  trainer  provided  the  classroom  instructor  in 
the  TTC  with  the  same  capabilities  as  were  provided  in  the  AW  “A”  and  STG  “A”  units.  The  TTC 
is  intended  to  bridge  the  gap  between  basic  Fleet  Replacement  Squadron  training  and  the  graduate 
level  Tactical  Training  Team  course.  It  is  one  of  five  requisite  courses  for  fulfilling  the  requirements 
to  become  a  PQS  qualified  TACCO  or  PPC. 

Objectives 

The  objectives  of  this  effort  are  (1)  to  evaluate  the  application  of  the  IMAT  system  in  the  STG 
“A”  school  with  respect  to  performance  on  factual,  comprehension,  and  cognitive  skill  test  items, 
student  motivation,  and  instructional  design;  (2)  to  compare  student  performance  on  STG  “A” 
IMAT  lessons  with  student  performance  on  comparable  Aviation  Systems  Warfare  Operator  (AW) 
“A”  school  IMAT  lessons;  (3)  to  evaluate  application  of  the  IMAT  system  in  the  TTC  at  Barbers 
Point;  and  (4)  to  compare  graduates  of  the  IMAT  TTC  with  qualified  fleet  tactical  officers  and  pilots 
on  tests  of  knowledge  of  acoustical  oceanography  and  tactical  problem  solving. 

Method 


STG  “A”  School  Evaluation 

Design 

The  STG  “A”  IMAT  application  was  evaluated  on  (1)  student  performance  on  three  types  of 
end  of  unit  test  items  (fact  items,  comprehension  items,  and  cognitive  skill  items),  (2)  student 
motivation,  and  (3)  quality  of  instructional  design.  STG  students  trained  with  the  IMAT  system 
were  compared  with  STG  students  trained  with  the  conventional  instruction  used  in  the  “A”  school 
prior  to  the  introduction  of  IMAT,  and  with  students  from  the  AW  “A”  who  had  also  been  trained 
with  the  IMAT  system. 

Subjects 

The  subjects  were  90  students  who  had  completed  the  STG  “A”  school  oceanography  unit  prior 
to  the  introduction  of  the  IMAT  system  (the  Standard  Instruction  group),  71  students  who  were 
taught  the  same  unit  with  IMAT  (the  IMAT  group),  and  46  students  who  completed  the  IMAT 
oceanography  unit  at  the  AW  “A”  school  (the  AW  IMAT  group).  Although  ASVAB  scores  were 
not  available  because  of  privacy  act  considerations,  the  STG  and  AW  “A”  schools  have  almost 
identical  ASVAB  entry  requirements.  Therefore,  the  student  samples  from  the  two  schools  were 
considered  to  be  equivalent  in  ability. 
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Instructional  Materials 


The  instructional  unit  for  all  three  groups  taught  acoustical  oceanography  and  all  groups  were 
compared  on  identical  end  of  unit  test  items.  The  major  topics  addressed  in  all  three  units  were 
environmental  factors  affecting  underwater  sound  transmission,  bathythermic  and  sound  velocity 
profiles,  sound  transmission  paths  including  ray  path  theoiy,  and  attenuation  and  spreading. 

Standard  Instruction.  The  number  of  learning  objectives  for  the  Standard  Instruction  is  five 
which  are  taught  in  26  class  periods  (approximately  4  and  a  half  days  including  testing  time).  The 
instructor  guide  is  20  pages  long  and  the  student  guide  consists  of  30  pages.  The  instruction  is  topic 
oriented  rather  than  teaching  the  knowledge  and  skills  in  a  job  context.  The  graphics  are  135  slides 
and  a  video  tape  on  the  deep  scattering  layer.  Neither  the  instructor  guide  nor  the  student  provide 
any  opportunities  for  practice  in  preparation  for  taking  the  unit  test. 

STG  IMAT  Instruction.  The  number  of  objectives  and  unit  length  for  the  STG  IMAT 
instruction  are  the  same  as  for  the  Standard  Instruction.  The  instructor  guide  is  50  pages  long  and 
20  pages  of  practice  exercises  (31  problems)  were  added  to  the  Standard  Instruction  30  page 
student  guide.  The  instruction  is  presented  in  the  context  of  the  job  of  detecting  and  tracking 
submarine  targets  under  a  variety  of  environmental  conditions.  The  graphics  are  193  IMAT 
displays  and  19  slides/overheads. 

AW  IMAT  Instruction.  The  number  of  objectives  for  the  portions  of  the  AW  unit  used  for  the 
comparison  is  59  and  the  length  is  39  class  periods  (approximately  6  and  a  half  days  including 
testing  time).  The  instructor  guide  is  205  pages  long  and  the  student  guide  is  190  pages  long  and 
includes  practice  for  each  of  the  objectives.  The  instruction  is  highly  contextualized.  The  graphics 
are  193  IMAT  displays  and  17  overheads. 

End  of  Unit  Test 

After  completing  their  respective  oceanography  units,  students  were  given  an  end  of  unit  test, 
which  included  items  that  could  be  used  to  compare  the  effects  of  STG-IMAT,  AW-IMAT,  and 
Standard  Instruction.  The  test  items  were  classified  into  one  of  the  following  categories  according 
to  the  type  of  information  tested  and  the  cognitive  processing  required  to  answer  the  question: 

Remember  Fact.  The  student  must  recall  or  recognize  names,  definitions,  steps  of  proce¬ 
dures,  formulas  an  terms  in  formulas,  labels  for  graphical  display,  or  technical  terminology. 

Remember  Qualitative  Information  (Comprehension).  The  student  must  recall  or  rec¬ 
ognize  cause  and  effect  relationships,  predictive  and  diagnostic  information,  or  how  and 
why  explanations  of  events  (e.g.,  diurnal  effect),  phenomena  (e.g.,  bottom  bounce),  and  de¬ 
vices  (e.g.,  sensors). 

Cognitive  Skills.  The  student  must  perform  a  sequence  of  steps,  including  making  deci¬ 
sions  and  judgments,  to  solve  a  problem,  calculate  or  determine  a  value  or  relationship,  or 
evaluate  a  scenario. 
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The  test  items  used  for  the  evaluation  were  32  Fact  items,  15  Qualitative  Information/ 
Comprehension  items,  and  16  Cognitive  Skill  items.  These  test  items  were  selected  from  a  pool  of  test 
items  developed  for  testing  oceanography  knowledge  and  skills.  All  test  items  in  the  pool  were 
reviewed  by  STG  “A”  and  AW  “A”  course  instructor  personnel  and  by  subject  matter  experts  assigned 
to  the  IMAT  project.  Items  that  were  rated  beyond  the  scope  of  the  instruction  or  inappropriate  for  “A” 
school  training  were  eliminated.  From  the  remaining  items,  the  63  used  in  this  evaluation  were  selected 
based  on  their  match  to  the  STG  “A”  and  AW  “A”  oceanography  unit  objectives.  Figure  1  displays  an 
example  of  each  type  of  item. 


Example  Fact  Item 

Sound  Channel  is  important  to  ASW  because: 

a.  all  sounds  in  the  Sound  Channel  portion  of  the  water  column  get 
trapped  for  long  distances 

b.  the  number  of  sounds  trapped  in  the  Sound  Channel  increase  to 
the  Limiting  Depth 

c.  limiting  rays  near  the  top  of  the  Sound  Channel  define  shadow 
zones  submarines  use  to  avoid  detection 

d.  the  top  of  the  Sound  Channel  traps  more  sounds  than  the  Sound 
Channel  Axis 

Example  Comprehension  Item 

The  probability  of  CZ  decreases  when  maximum  near-surface  velocity: 

a.  adjusts  to  a  cooler  surface  layer 

b.  decreases 

c.  lowers  the  limiting  depth 

d.  occurs  higher  in  the  water  column 

_ _ Figure  1.  Example  test  items. _ 


Example  Cognitive  Skill  Item 

Look  at  the  following  SVP  and  ocean  description.  Note  that  depth  is  indicated  in  feet. 
Which  sound  transmission  paths  are  possible  for  this  water  column  with  these 
environmental  characteristics?  Answer  the  questions  by  selecting  “T”  to  indicate  possible 
transmission  paths  and  the  letter  “F’  for  paths  that  are  not  possible. 

4600  4800  5100 


Sea  state: 

2 

Shipping  intensity 

low 

Bottom  Bounce  (BB) 

Convergence  Zone  (CZ) 

Direct  Path  ODP) 

Half  Channel  (HC) 

Sound  Channel  (SC) 

Surface  Duct  (SD) 

Figure  1.  (Continued). 
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Motivational  Analyses 

For  the  motivational  analysis  a  questionnaire  based  on  the  Attention-Relevance-Confidence- 
Satisfaction  (ARCS)  motivational  assessment  model  originated  by  Keller  (1992)  was  developed.  The  34 
item  questionnaire,  designed  for  application  to  technical  training,  was  administered  to  all  of  the  71  STG 
“A”  school  IMAT  students.  Unfortunately,  the  motivation  questionnaires  for  the  Standard  Instruction 
students  were  either  not  collected  or  were  lost  in  transit  and  could  not  be  analyzed.  In  a  previous  study 
(Ellis  &  Parchman,  1994),  the  ARCS  questionnaire  was  administered  to  76  AW  “A”  school  students 
trained  with  the  IMAT  system.  These  data  were  used  in  the  present  evaluation  for  comparison.  The  ARCS 
questionnaire  assess  four  motivational  characteristics;  attention,  confidence,  relevance,  and  satisfaction. 
The  attention  oriented  questions  assess  how  well  the  material  captures  the  interest  of  the  learners  and 
stimulates  their  curiosity  to  learn.  The  relevance  questions  address  how  well  the  materials  meet  the  needs 
and  goals  of  the  learner.  The  confidence  questions  ask  the  students  to  report  on  their  beliefs  and  feelings 
about  how  well  they  will  succeed  and  how  much  they  can  control  their  own  success.  The  satisfaction 
questions  concern  students  feelings  of  reward  and  accomplishment  and  their  enjoyment  of  the  materials. 
The  four  scales  are  scored  from  1  to  5  with  1  indicating  a  statement  about  the  course  is  “Not  true”  and  a 
5  indicating  a  statement  is  “Very  true.”  A  score  of  3  indicates  students  believe  a  statement  is  “Moderately 
true.” 

Instructional  Design  Evaluation 

The  Course  Evaluation  System  (CES)  (Ellis,  Knirk,  Taylor,  &  McDonald,  1993)  was  used  to  compare 
the  instructional  design  of  the  IMAT  and  Standard  Instruction  oceanography  units.  In  a  previous  study 
(Ellis  &  Parchman,  1994),  the  CES  was  used  to  evaluate  the  AW  “A”  IMAT  unit.  Relevant  portions  of 
these  data  were  used  for  comparisons  with  the  STG  units  in  the  present  evaluation.  The  CES  assesses  the 
consistency  (match)  among  learning  objectives,  test  items,  and  the  instructional  presentation,  and  the 
adequacy  of  the  instructional  presentation.  For  the  present  evaluation,  however,  the  objectives  and  test 
items  for  the  two  STG  units  were  identical.  For  the  AW  unit,  the  objectives  were  different  but  the  test 
items  were  identical.  Therefore,  the  CES  was  used  to  assess  the  consistency  among  the  test  items  and  the 
instructional  presentations  for  each  unit.  The  consistency  assessments  were  then  compared  for  the 
evaluation.  The  consistency  assessment  employed  the  same  classification  scheme  for  test  items  used  for 
the  end  of  unit  test. 

Tactical  Training  Course  Evaluation 

Design 

For  the  TTC  evaluation  students  were  administered  a  paper  and  pencil  knowledge  and  cognitive  skills 
test  upon  entering  the  school  and  after  completing  the  course  in  a  standard  pre-post  design.  The  scores  of 
TTC  graduates  were  also  compared  with  the  performance  of  qualified  fleet  tacticians  and  pilots  on  the 
same  testing  instrument. 

Subjects 

Subjects  were  59  non-Personnel  Qualification  System  (PQS)  qualified  officers  who  were  selected  to 
enroll  in  the  TTC  course.  Of  this  group  26  were  tacticians  and  33  were  pilots.  The  fleet  comparison 
subjects  were  21  PQS  qualified  officers.  Of  this  group  9  were  TACCOs  and  12  were  PPCs. 
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Instructional  Materials 


The  TTC  is  a  10  day  course  in  which  tactical  oceanography  is  one  of  several  topics.  The 
number  of  learning  objectives  for  the  tactical  oceanography  portion  of  TTC  course  is  19,  which  are 
taught  over  8  and  a  half  class  periods  spread  over  3  instructional  days.  The  instructor  guide  is  40 
pages  long  and  the  student  notebook  is  42  pages.  The  instruction  is  presented  in  the  context  of  the 
job  of  locating,  tracking,  and  prosecuting  submarine  targets  under  a  variety  of  environmental 
conditions  and  tactical  situations.  The  graphics  are  120 IMAT  displays. 

Pre-  and  Post-Course  Tests  and  Testing  Procedures 

The  pre-  and  post-course  tests  were  designed  to  assess  application  of  environmental  principles 
and  tactical  planning.  Two  parallel  fonns  of  the  test  were  developed  for  the  pre-post  design.  The 
pretest  contained  17  knowledge  question  that  included  both  fact  and  comprehension  items  and  7 
scenario  problems  that  tested  cognitive  skills.  The  posttest  contained  16  different  knowledge 
questions  and  the  same  7  scenario  problems  that  were  on  the  pretest.  Figure  2  presents  a  sample 
scenario  problem.  All  test  items  were  reviewed  and  validated  by  subject  matter  specialists.  The 
pretest  was  given  to  students  prior  to  taking  the  TTC  course.  The  posttest  was  given  to  students 
upon  completion  of  the  tactical  oceanography  portion  of  the  course  and  also  to  the  qualified  fleet 
pilots  and  tacticians.  In  addition,  all  subjects  were  given  a  demographic  questionnaire  to  determine 
years  of  experience,  years  of  PQS  qualification  (for  fleet  personnel),  age,  and  years  in  service. 
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You  are  on  deplo3Tnent  in  Adak,  Alaska,  in  the  winter.  Your  first  task  to  search,  localize  and  track 
an  SSBN  operating  in  the  North  Pacific 


Trace  A 


Trace  B 


Trace  C 


Temperature 

40  50  60  70 


Temperature 

40  50  60  70 


Temperature 

40  50  60  70 


£ 

§1000 

O 


BT  PROFILE 


BT  PROFILE 


BT  PROFILE 


Which  of  the  above  BT  traces  (trace  A,  trace  B,  or  trace  C)  would  you  expect  to  find  around  60  North 
Latitude  (in  your  operating  area),  in  the  winter?  Answer:  BT  Trace _ 

Draw  the  Sound  Speed  Profile  that  corresponds  to  the  BT  Trace  that  you  selected  and  Label  your  SSP 
drawing  with  the  sound  transmission  pathways  (i.e.,  surface  duct,  deep  sound  channel,  etc.) 

Your  Sound  Speed  Profile 


Depth 


Speed _ 


Figure  2.  Sample  scenario-based  problem. 
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Results  and  Discussion 


STG  “A”  School  Evaluation 
End  of  Unit  Test  Analyses 

A  3  Groups  (STG  Standard  Instruction  vs.  STG  IMAT  vs.  AW  IMAT)  by  3  Item  Type  (Fact  vs. 
Comprehension  vs.  Cognitive  Skill)  analysis  of  variance  (ANOVA)  was  performed  on  percent 
correct  for  each  type  of  item  on  the  end  of  unit  test.  Table  1  presents  the  means  for  this  analysis. 
Both  main  effects  and  the  Groups  by  Item  Type  interaction  were  significant  (p  <  .01).  Post  hoc 
Tukey  HSD  comparisons  revealed  that  AW  IMAT  students  scored  higher  on  all  item  types  than  the 
other  two  groups  (p  <  .05).  Further,  the  STG  MAT  students  scored  higher  on  all  item  types  than 
the  STG  Standard  Insti’uction  students  (p  <  .05).  The  significant  interaction  was  attributable  to 
students  in  both  the  IMAT  groups  scoring  relatively  better  on  Cognitive  Skill  and  Comprehension 
items  compared  to  Fact  items  than  Standard  Instruction  students  (p  <  .05). 

Table  1 

Mean  Percent  Correct  for  Fact,  Comprehension,  and  Cognitive  Skill  Test 
Items  for  STG-IMAT,  AW-IMAT,  and  Standard  Instruction  Groups 


Facts  (%) 
N=32  Items 

Comprehension 

(%) 

N^15  Items 

Cognitive  Skills 
(%) 

N=  16  Items 

Total  (%) 

=  63  Items 

STG  Standard  Instruction 
iV  =  90 

79.5 

63.2 

62.4 

71.3 

STG  IMAT 

N  =  l\ 

89.0 

76.0 

111 

82.9 

AW  IMAT 

N  =  A6 

94.9 

87.8 

83.8 

90.4 

The  finding  that  MAT  students  performed  better  on  all  three  types  of  test  items  supports  the 
hypothesis  that  the  combination  of  strategies  incorporated  in  the  MAT  system  provides  an 
effective  environment  for  learning  complex  knowledge  and  skills.  The  most  important  result  for 
the  hypothesis  is  that  both  STG-  and  AW-IMAT  students  performed  better  on  Comprehension  and 
Cognitive  Skill  items  relative  to  Fact  items  than  the  Standard  Instruction  students.  That  is,  the 
significant  Groups  by  Item  Type  interaction  shows  that  the  MAT  system  differentially  increased 
learning  and  performance  for  the  more  complex  types  of  items.  This  result  is  exactly  what  MAT 
was  designed  to  accomplish  and  directly  addresses  the  post  Cold  War  training  requirements  for 
sensor  operators. 

Instructional  Design  Analyses 

The  significant  differences  among  the  three  groups  on  the  three  types  of  test  items  can  in  part 
be  explained  by  comparing  the  degree  to  which  some  of  the  fundamental  instructional  principles 
underlying  the  MAT  system  were  applied  in  each  unit.  The  results  of  the  CES  analysis  reveals 
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some  of  these  differences.  First,  the  total  number  of  objectives  for  the  AW-IMAT  unit  was  59 
versus  5  for  the  STG  units  (remember  that  course  length  and  objectives  did  not  change  for  the  STG 
IMAT  application).  Second,  the  AW  unit  length  was  39  class  periods  versus  26  for  the  STG  units. 
For  the  AW  unit,  33  objectives  were  Fact,  19  were  Comprehension,  and  7  were  Cognitive  Skill. 
For  the  STG  units  four  objectives  were  Fact,  one  was  Cognitive  Skill,  and  none  were 
Comprehension.  Third,  the  number  of  graphic  displays  increased  by  nearly  50  percent  for  both 
IMAT  groups. 

The  differences  in  length  and  number  and  type  objectives  result  in  part  from  the  AW-IMAT  unit 
being  completely  redesigned,  while  the  STG-IMAT  unit  was  only  modified  to  accommodate 
IMAT.  The  AW-IMAT  redesign  reflects  a  thorough  front-end  analysis  of  the  knowledge  and  skills 
that  are  required  for  proficiency  and  a  concern  for  teaching  the  cognitive  concepts  and  relationships 
needed  to  reason  in  a  complex  environment.  These  major  differences  in  unit  length,  number  and 
type  of  objectives,  depth  of  analysis,  and  training  emphasis  probably  all  work  to  improve  the 
performance  of  AW-IMAT  students.  Additionally,  the  increased  number  of  displays  and  the 
accompanying  contextualized  elaborated  explanations  may  have  contributed  to  the  difference 
between  the  two  STG  groups.  However,  the  increase  in  displays  alone  does  not  account  for  the 
performance  differences  between  AW-  and  STG-IMAT  students  as  the  displays  used  in  each  unit 
were  identical  (n  =193).  In  the  AW  course  the  displays  were  more  highly  integrated  with  the 
elaborated  examples  and  practice  than  in  the  STG  IMAT  course,  which  only  incorporated  portions 
of  the  AW  student  and  instructor  guides.  Thus,  the  combined  effects  of  the  displays  and 
accompanying  instructional  approach  could  have  produced  the  performance  differences. 

Although  perfonnance  improvements  were  predicted  for  the  IMAT  groups,  the  increased 
performance  on  Fact  items  for  both  IMAT  groups  compared  to  the  Standard  Instruction  group 
shows  a  benefit  for  IMAT  training  that  was  not  observed  in  the  results  of  the  Ellis  and  Parchraan 
(1994)  evaluation  of  the  AW-IMAT  unit.  They  found  no  differences  between  AW-IMAT  students 
and  AW  previous  unit  (standard  instniction)  students  on  comparable  multiple-choice  items  that 
tested  factual  information.  This  discrepancy  can  be  accounted  for  by  examining  the  opportunities 
for  practice  in  each  unit.  The  AW-IMAT,  AW  previous,  and  STG-IMAT  units  all  provided  some 
practice  opportunities  for  the  Fact  objectives,  while  the  STG  Standard  Instruction  had  no  such 
opportunities.  Further,  the  AW-IMAT  provided  more  opportunities  for  Fact  practice  than  the  STG- 
IMAT.  Practice  opportunities  may  also  in  part  account  for  the  differences  between  STG-IMAT  and 
Standard  Instruction  students  on  Comprehension  and  Cognitive  Skill  items.  Again,  there  was  no 
practice  for  Standard  Instruction  students  while  31  problems  that  included  practice  on  all  three 
types  of  items  were  added  to  the  STG-IMAT  unit  (see  Table  2). 

The  CES  evaluation  of  the  consistency  among  instructor  and  students  guides  and  the  test  items 
also  reveals  differences  among  the  three  units  that  may  account  for  the  performance  differences. 
Table  2  shows  percent  of  instructional  components  that  are  present  in  the  instructor  and  student 
guides  for  each  type  of  test  item  for  each  unit.  For  example,  if  there  were  10  Fact  test  items,  each 
item  would  require  a  statement  and  an  opportunity  to  practice  remembering  to  be  present 
somewhere  in  the  instructor  or  student  guides.  If  there  were  only  five  statements  and  three  practice 
questions  for  the  10  Fact  items,  the  percentages  in  the  Statement  and  Practice  Remembering  w/ 
Feedback  columns  would  be  50%  and  30%,  respectively.  (Note  that  for  Fact  and  Comprehension 
items  the  Example  and  Practice  Using  components  are  not  required).  In  the  AW  IMAT  course  all 
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required  components  are  present,  while  components  are  present  to  a  lesser  degree  for  the  two  STG 
units.  In  the  STG  Standard  Instruction,  the  practice  component  for  all  three  categories  of  test  item 
is  not  present  at  all.  This  more  detailed  analysis  confirms  the  general  analysis  of  practice 
opportunities  just  discussed.  The  absence  of  practice  and  other  required  instructional  components 
for  specific  test  items  in  the  Standard  Instruction  and  to  a  lesser  degree  in  the  STG-IMAT 
instruction  may  account  for  some  of  the  performance  differences  among  the  three  groups. 

Table  2 

Percent  of  Each  Required  Presentation  Component  that  is  Present  for  Each  Type  of 
Test  Item  for  the  STG-IMAT,  AW-IMAT,  and  Standard  Instruction  Units 


Required  Presentation  Components 
Practice 


Test  Item  Type  and 
Instructional  Unit 

Statement 

Remembering 

w/Feedback 

Examples 

Practice  Using 
w/Feedback 

Fact(Af=32) 

Standard  Instruction 

75.0 

0.0 

NA 

NA 

STG-IMAT 

90.6 

18.6 

NA 

NA 

AW-IMAT 

100.0 

100.0 

NA 

NA 

Comprehension  (A^=  15) 

Standard  Instruction 

66.7 

0.0 

NA 

NA 

STG-IMAT 

100.0 

40.0 

NA 

NA 

AW-IMAT 

100.0 

100.0 

NA 

NA 

Cognitive  Skill  (N  =  16) 

Standard  Instruction 

12.5 

0.0 

6.3 

0.0 

STG-IMAT 

37.5 

87.5 

100.0 

87.5 

AW-IMAT 

100.0 

100.0 

100.0 

100.0 

Motivational  Analysis 

The  ARCS  questionnaire  assesses  four  motivational  characteristics;  attention,  confidence, 
relevance,  and  satisfaction.  Each  characteristic  is  rated  from  one  to  five;  not  true  to  very  true, 
respectively.  The  mean  scores  for  each  scale  for  the  STG-IMAT  unit  are:  Attention  =  3.38, 
Confidence  =  3.77,  Relevance  =  3.79,  and  Satisfaction  =  3.47.  These  scores  are  not  significantly 
different  from  the  ARCS  scores  for  AW  “A”  IMAT  students  collected  in  a  previous  study  (Ellis  & 
Parchman,  1994).  The  AW  scores  were  Attention  =  3.43,  Confidence  =  3.35,  Relevance  =  3.76,  and 
Satisfaction  =  3.49.  As  reported  in  Ellis  and  Parchman  (1994),  motivation  scores  in  the  ranges 
obtained  in  both  the  AW  and  STG  IMAT  units  are  significantly  higher  than  scores  obtained  for  both 
standard  instruction  and  specially  designed  individualized  computer  based  training  in  an 
introductory  electricity  course. 
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Tactical  Training  Course  Evaluation 
Pre-Post  Analysis 

Scores  for  pilots  and  tacticians  on  the  pre  and  posttests  for  the  knowledge  and  scenario  test 
items  were  compare  using  t-tests.  There  were  significant  gains  from  pretest  to  posttest  for  both 
types  of  items  for  both  pilots  and  tacticians  (p  <  .01).  Table  3  presents  the  mean  percent  correct  for 
the  knowledge  and  scenario  items  for  pilots  and  tacticians.  This  result  shows  that  the  IMAT  system 
was  effective  in  achieving  the  course  goals. 


Table  3 

Mean  Percent  Correct  for  Knowledge  and  Scenario  Items  on  the 
TTC  Pre-  and  Posttest  for  Pilot  and  Tacticians 


Pretest 

Posttest 

Knowledge  Items 

Tacticians 

55.4 

81.6 

Pilots 

47.9 

82.0 

Scenario  Items 

Tacticians 

54.7 

78.3 

Pilots 

48.2 

76.7 

Totals 

Tacticians 

55.1 

79.7 

Pilots 

47.4 

78.2 

Fleet  Comparison  Analysis 

A  2  job  category  (pilot  vs.  tactician)  by  2  item  type  (scenario  vs.  knowledge)  by  2  experience 
(TTC  graduate  vs.  PQS  qualified)  ANOVA  was  performed  on  the  scores  on  the  knowledge  and 
scenario  items  on  the  TTC  end  of  course  test.  The  only  significant  difference  was  for  experience 
with  TTC  graduates  scoring  higher  than  PQS  qualified  fleet  personnel  on  both  scenario  and 
knowledge  items.  Table  4  presents  the  means  for  this  analysis.  The  finding  that  TTC  students  score 
higher  than  fleet  qualified  personnel  shows  that  IMAT  can  accelerate  the  development  of  expertise 
on  job  relevant  scenarios  as  well  as  provide  an  effective  cognitive  context  for  assimilating  job 
knowledge.  Retesting  of  TTC  graduates  is  planned  after  they  are  PQS  qualified  to  determine  the 
long  term  effects  of  IMAT  training. 
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Table  4 


Mean  Percent  Correct  for  Knowledge  and  Scenario  Items  for 
TTC  and  Fleet  Qualified  Pilots  and  Tacticians 


TTC  Graduates  PQS  Qualified  Fleet  Personnel 


Knowledge  Items 


Tacticians 

81.6 

72.9 

Pilots 

82.0 

71.9 

Scenario  Items 

Tacticians 

78.3 

73.8 

Pilots 

76.7 

69.6 

Totals 

Tacticians 

79.7 

73.5 

Pilots 

78.2 

70.4 

Conclusions 

Several  conclusions  can  be  drawn  from  these  evaluations.  First,  research  on  cognition  and 
instruction  and  technological  advances  in  scientific  visualization  can  be  integrated  and  applied  in 
real  world  training  to  produce  substantial  gains  in  performance  and  student  motivation.  Second, 
the  IMAT  system  has  achieved  its  intended  design  goals  by  effectively  teaching  complex 
knowledge  and  cognitive  skills.  Third,  the  IMAT  system  emphasis  on  inclusion  of  required 
instructional  components,  especially  practice  opportunities,  contributed  significandy  to  the 
observed  performance  improvements.  Finally,  the  TTC  findings  show  that  IMAT  can  accelerate  the 
development  of  expertise  in  tactical  oceanography  and  decision  making.  Overall,  the  results  of  the 
STG  and  TTC  implementations  show  that  IMAT  is  a  highly  effective  training  system  that  offers  a 
viable  solution  for  many  of  the  training  requirements  and  challenges  faced  by  the  four  ASW 
communities  in  the  post  Cold  War  world. 
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